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PROJECT  OBJECTIVES  AND  SUMMARY 

Because  the  military  is  increasingly  relying  on  geographically  distributed  systems  using 
technology-mediated  interaction,  it  is  critical  that  we  understand  the  cognitive  processes  affected 
by  distributed  interaction  (at  the  inter-  and  intra-individual  level),  the  specific  conditions 
affecting  them,  and  how  training  can  be  best  delivered  for  such  environments.  Given  the 
impending  need  to  better  understand  these  issues,  our  ultimate  objective  is  to,  not  only 
understand  the  complex  nature  of  distributed  training  for  Air  Force  operations,  but  also  identify 
the  support  tools  needed  to  optimize  training  designed  for  distributed  systems.  Towards  that  end, 
our  objectives  for  this  3-year  project  revolve  around  investigation  of: 

1)  The  feasibility  and  utility  of  cognitively  diagnostic  assessment  in  distributed  training; 

2)  The  identification  of  the  learning  outcomes  associated  with  distributed  training  and  the 
individual  differences  which  may  impact  these  outcomes;  and, 

3)  The  types  of  system  augmentation  techniques  that  best  mediate  training  success  when 
presented  with  differing  task  and  individual  characteristics  and  delivered  on  distributed 
training  systems. 

We  have  maintained  two  simultaneous  thrusts  with  this  effort.  Our  first  thrust  has  involved 
theoretical  development  so  that  we  may  capture  the  foundational  elements  associated  with 
distributed  training  and  distributed  teams.  Our  second  thrust  has  involved  empirical  examination 
of  the  complex  mix  of  factors  associated  with  distributed  training.  In  short,  we  have  explored  the 
fundamental  issues  in  knowledge  acquisition  and  cognitively  diagnostic  assessment  to  determine 
how  to  best  train  the  knowledge  foundational  to  distributed  training.  This  has  allowed  us  to  gain 
a  better  understanding  of  how  knowledge  acquisition  is  related  to  knowledge  application  in  the 
context  of  distributed  training  and  distributed  teams. 

Our  overall  goal  is  to  provide  the  foundation  on  which  we  can  design  tools  capable  of  diagnosing 
competencies  associated  with  complex  task  performance  for  individuals  and  teams,  but 
deliverable  via  distributed  training  systems.  In  this  brief  report  we  provide  an  overview  of  our 
project.  We  first  describe  our  accomplishments  in  Year  One  that  formed  the  foundation  for  our 
subsequent  efforts  and  follow  this  with  a  brief  review  of  our  objectives  and  accomplishments  for 
Year  Two  and  Year  Three.  Based  upon  these  findings  and  accomplishments,  we  then  provide  a 
summary  of  the  benefits  of  this  research  to  the  overall  AFOSR  Mission.  Due  to  page  limitations, 
we  only  abstract  our  work  in  this  document.  For  full  reprints  of  all  described  manuscripts 
produced  with  this  grant,  please  contact,  the  Principal  Investigator,  Dr.  Eduardo  Salas. 
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THEORETICAL  AND  PRACTICAL  ISSUES  ASSOCIATED  WITH  ADL  RESEARCH 


Year  One  -Accomplishments  and  Key  Findings 

1 .  We  first  identified  a  testbed  appropriate  for  investigation  of  the  factors  critical  to  distributed 
training  and  distributed  teams.  Specifically,  our  goal  was  to  identify  a  testbed  capable  of 
eliciting  and  capturing  individual  and  team  behaviors  associated  with  human  information 
processing  and  resulting  individual  and  team  decision  making.  Thus,  we  focused  on 
identifying  the  research  tool  that  would  provide  the  environment  to  investigate  individual  and 
team  cognition  along  with  critical  team  behaviors.  We  adopted  a  complex  distributed  team 
training  testbed  (DDD  synthetic  task)  specifically  designed  as  a  low-fidelity  simulation  that 
allows  basic  issues  to  be  investigated  in  a  complex,  yet  controlled,  setting. 

2.  We  next  designed,  developed,  and  tested  measurement  protocols  within  the  context  of  the 
chosen  testbed  (i.e.,  the  DDD  synthetic  task).  Specifically,  to  strengthen  the  base  on  which 
we  would  investigate  distributed  training,  our  goal  was  to  devise  measures  designed  to  tap 
individual  and  team  task-relevant  knowledge. 

3.  We  additionally  developed  a  laboratory  environment  capable  of  conducting  research  in 
distributed  learning  environments.  Specifically,  our  “Consortium  for  Research  in  Adaptive 
Distributed  Learning  Environments”  (CRADLE)  addresses  the  increasingly  critical  needs 
associated  with  instruction  in  adaptive  distributed  learning  environments  (ADL).  Through 
partnerships  with  multiple  research  institutions,  our  overall  goal  was  to  create  an 
environment  that  allows  us  to  develop  a  fuller  understanding  of  the  learning  process  when 
ADL  environments  are  utilized. 

4.  In  addition  to  meeting  the  aforementioned  objectives  we  also  completed  the  following:  (1) 
conducted  a  review  of  distributed  learning  research  that  identifies  the  emerging  themes  in 
distance  and  distributed  learning;  and,  (2)  developed  a  theoretical  framework  of  knowledge 
acquisition  in  adaptive  distributed  learning  environments. 
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Year  One  -  Key  Findings  and  Relevant  Products 
Theoretical  Work 


Salas,  E.,  Kosarzycki,  M.  P.,  Burke,  C.  S.,  Fiore,  S.  M.,  &  Stone,  D.  L.  (2002).  Emerging 
themes  in  distance  learning  research  and  practice:  Some  food  for  thought.  International 
Journal  of  Management  Reviews,  4, 135-153. 

With  the  rapid  spread  of  distance  learning  as  a  medium  for  delivering  instruction,  the  practice  of 
distance  learning  has  outpaced  research.  This  article  describes  major  themes  identified  in  a 
review  of  selected  research  articles  published  in  the  past  five  years.  Themes  include  the 
following:  definitions  of  distance  learning  and  why  it  should  be  studied,  identification  of  the 
major  learning  theories  on  which  research  is  based,  how  collaboration  can  be  achieved  via 
distance  learning,  the  role  that  learner  characteristics  play  in  the  success  of  distance  learning 
systems,  and  issues  related  to  measuring  the  effectiveness  of  distance  learning.  The  authors 
conclude  that  more  research  is  needed  to  identify  critical  success  factors  for  distance  learning. 

Empirical  Work 

Fiore,  S.  M.,  Cuevas,  H.  M.,  Scielzo,  S.,  &  Salas,  E.  (2002).  Training  individuals  for  distributed 
teams:  Problem  solving  assessment  for  distributed  mission  research.  Computers  in 
Human  Behavior,  18, 125  -  140. 

In  this  paper  we  describe  an  effort  investigating  the  feasibility  and  utility  of  cognitively 
diagnostic  assessment  of  problem  solving  when  training  for  distributed  team  tasks.  We  utilized 
computer-based  knowledge  elicitation  methods  to  assess  both  relational  problem  solving, 
requiring  the  semantic  integration  of  concepts,  and  dynamic  problem  solving,  requiring  the 
ability  to  integrate  and  apply  these  concepts.  Additionally,  we  addressed  how  metacognitive 
processes  interact  with  learning  outcomes  when  training  for  complex  synthetic  task 
environments.  We  find  first,  that  multiple  methods  of  assessing  problem  solving  performance 
are  diagnostic  of  knowledge  acquisition  for  a  complex  synthetic  team  task,  and  second,  that 
general  metacomprehension  predisposition  is  related  to  metacomprehension  accuracy  in 
synthetic  task  environments. 

Supplemental  Publications 

Fiore,  S.  M.,  Salas,  E.,  &  Bowers,  C.  A.,  (2001).  Attitudes,  behaviors,  and  cognition  in 
distributed  teams:  The  effect  of  team  opacity  on  process  and  performance.  Proceedings 
of  the  45th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society,  382-386. 
Santa  Monica,  CA:  Human  Factors  and  Ergonomics  Society. 
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Year  Two  -  Objectives  and  Accomplishments 

Following  our  foundational  efforts  in  Year  One,  in  Year  Two  our  objectives  involved  theoretical 
and  empirical  thrusts  centered  on  distributed  learning  and  training  for  complex  team  tasks.  The 
objective  for  our  empirical  thrust  involved  understanding  the  instantiation  and  development  of 
the  knowledge  structures  for  complex  tasks  when  distributed  learning  systems  are  utilized.  More 
specifically,  the  issue  is  how  knowledge  is  integrated,  a  key  component  of  the  development  of 
expertise  (see  Glaser,  1989).  We  have  pursued  this  so  that  we  could  understand  how  to  train  for 
knowledge  integration  at  the  individual  level  and  the  team  level  (e.g.,  team  coordination). 
Simultaneous  to  this  we  have  been  investigating  the  individual  differences  that  may  mediate 
these  factors.  This  allows  us  to  determine  the  predictive  utility  of  particular  aptitudes  and  how 
they  may  influence  learning  and  performance  for  complex  team  tasks. 

To  explore  this  issue  our  empirical  work  has  involved  investigations  of  the  issues  outlined  in  our 
Constructive  Cognitive  Activities  Framework  (Figure  1),  specifically  studies  of: 

(1)  The  manner  in  which  multi-sensory  training  alters  cognitive  processes; 

(2)  How  individual  differences  impact  and  are  impacted  by  learning  in  complex  task 
environments; 

(3)  The  methods  necessary  for  knowledge  acquisition  in  complex  tasks. 

(4)  The  development  and  evaluation  of  a  testbed  designed  to  train  team  competencies  for  a 
complex  distributed  team  task;  one  that  trains  strategies  and  metacognitive  processes. 

(5)  The  testing  of  a  working  memory  battery  developed  by  the  AFOSR  “LAMP”  program. 
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Figure  1.  Constructive  cognitive  activities  framework. 
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To  meet  the  objectives  of  our  theoretical  thrust  we  have  proceeded  along  two  tracks,  one  focused 
on  the  present  day  issues  with  respect  to  distributed  training,  and  the  other  focused  more  towards 
the  future  and  the  process  issues  associated  with  team  interaction. 

(1)  The  first  portion  of  our  theoretical  work  involved  an  investigation  of  distributed  learning  as  it 
is  understood  and  used  in  organizations  today  (Kosarzycki,  Salas,  DeRoun,  &  Fiore,  2003) 
along  with  the  preparation  of  an  edited  volume  on  Adaptive  Distributed  Learning 
environments  (Fiore  &  Salas,  in  preparation). 

(2)  The  second  portion  of  our  work  involved  the  development  of  an  understanding  of  team 
process  and  performance  associated  with  team  interaction  in  general  (Salas  &  Fiore,  2004) 
and  distributed  interaction  in  particular  (Cuevas,  Fiore,  Bowers,  &  Salas,  2004;  Fiore,  Salas, 
Cuevas,  &  Bowers,  2003). 
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Year  Two  -  Key  Findings  and  Relevant  Products 
Theoretical  Work 

Salas,  E.  &  Fiore,  S.  M.  (Eds.)  (2004).  Team  cognition:  Understanding  the  factors  that  drive 
process  and  performance.  Washington  DC:  American  Psychological  Association. 

The  purpose  of  this  volume  is  to  provide  an  avenue  with  which  researchers  from  a  diverse  array 
of  fields  can  present  their  theories  and  findings  on  team  cognition  in  one  integrated  volume.  As 
such,  chapters  were  written  by  a  recognized  leader  within  their  own  discipline  and  allowed  them 
to  discuss  their  research  in  the  context  of  a  broad-based  approach.  Thus,  this  book  allows 
researchers  from  differing  disciplines  to  be  brought  up-to-date  on  both  theoretical  and 
methodological  issues  surrounding  team  cognition.  Our  overall  goal  with  this  volume  is  to 
illuminate  team  cognition  through  discussion  of  the  many  approaches  that  have  been  utilized  in 
order  to  converge  on  a  better  understanding  of  this  relatively  amorphous  concept. 

Fiore,  S.  M.,  Salas,  E.,  Cuevas,  H.  M.  &  Bowers,  C.  A.  (2003).  Distributed  coordination  space: 
Toward  a  theory  of  distributed  team  process  and  performance.  Theoretical  Issues  in 
Ergonomic  Science,  4,  3-4,  340-363. 

In  this  paper  we  discuss  a  theoretical  framework  designed  to  elucidate  the  many  issues 
surrounding  distributed  team  performance,  emphasizing  how  work  characteristics  associated 
with  such  teams  may  alter  both  the  processes  and  the  products  emerging  from  distributed 
interaction.  We  suggest  that  distributed  team  performance  can  best  be  understood  through 
conceptualization  of  a  coordination  space  within  which  distributed  interaction  occurs  over  time 
and  distance.  Our  goal  is  to  take  a  socio-technical  approach  to  distributed  team  research  so  that 
we  can  explicate  both  the  cognitive  consequences  of  a  lack  of  co-location  as  well  as  the  social 
consequences  affecting  interaction  and  team  development  when  work  is  technology-mediated. 
Our  overall  objective  is  to  present  a  framework  of  “distributed  coordination”  such  that  the 
principles  most  appropriate  for  distributed  team  performance  can  be  developed. 

Cuevas,  H.  M.,  Fiore,  S.  M.,  Salas,  E.,  &  Bowers,  C.  A.  (2004).  Virtual  teams  as  socio-technical 
systems.  In  S.  H.  Godar  &  P.  Ferris  (Eds.),  Virtual  and  collaborative  teams:  Process, 
technologies,  and  practice  (pp.  1-19). 

In  this  chapter,  we  adopt  a  sociotechnical  systems  approach  to  understand  the  challenges  faced 
by  members  of  an  organizational  unit  that  is  not  constrained  by  geographical,  temporal, 
organizational,  and/or  national  boundaries.  Specifically,  we  examine  distributed  team 
performance  within  the  context  of  an  open  sociotechnical  system,  highlighting  the  effects  that  the 
technological  subsystem  (e.g.,  collaborative  information  technology)  and  external  environmental 
factors  (e.g.,  lack  of  co-location)  have  on  the  personnel  subsystems  (i.e.,  distributed  team 
members)  within  the  organization.  The  organizational  psychology  literature  on  group 
productivity,  motivation,  and  shared  mental  models  is  reviewed  to  understand  team  performance 
within  the  context  of  distributed  environments,  and  in  order  to  offer  guidelines  and  interventions 
for  organizational  practice. 
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Kosarzycki,  M.  P.,  Salas,  E.,  DeRouin,  R.,  &  Fiore,  S.  M.  (2003).  Distance  learning  in 
organizations:  A  review  and  future  needs.  In  D.  Stone  (Ed.),  Advances  in  human 
performance  and  cognitive  engineering  research,  Volume  3  (pp.  69  -  98).  Amsterdam: 
JAI  Press,  Elsevier  Science. 

Many  organizations  have  implemented  distance-learning  (DL)  courses  and  programs  as  an 
economical,  efficient  way  to  deliver  training.  The  purpose  of  this  chapter  is  to  summarize  some 
of  the  major  considerations  that  are  associated  with  distance-learning  programs.  We  describe  a 
number  of  the  issues  surrounding  DL,  ranging  from  how  organizations  use  DL  to  the  differing 
forms  of  training  being  delivered  and  how  organizations  are  reacting  to  DL.  We  close  with  a 
discussion  of  issues  in  practice  and  suggest  directions  for  future  research. 


Empirical  Work 

Fiore,  S.  M.,  Cuevas,  H.  M.,  Scielzo,  S.,  &  Salas,  E.  (in  press).  Metacognitive  bias  in  complex 
task  training:  Influence  of  verbal  ability  in  performance  prediction  accuracy.  To  appear 
in  S.  M.  Fiore  &  E.  Salas  (Eds.),  Where  is  the  learning  in  distance  learning?:  Towards  a 
science  of  distributed  learning  and  training.  Washington  DC:  American  Psychological 
Association. 

In  this  chapter  we  address  how  metacognitive  processes  interact  with  learning  outcomes  when 
training  for  complex  task  environments.  We  report  the  findings  from  two  different  studies  that 
show  strikingly  similar  patterns  of  performance  with  respect  to  differences  in  verbal  ability, 
metacognition,  and  performance.  Participants’  metacognitive  bias  scores  (differences  between 
prediction  and  performance)  were  negatively  related  to  performance  and  the  difference  in  bias 
scores  between  high  and  low  verbal  ability  participants  was  greater  with  more  complex  measures 
of  knowledge  acquisition.  Furthermore,  these  bias  scores  consistently  revealed  that  low  verbal 
ability  participants  were  overestimating  their  performance  while  high  verbal  ability  participants 
were  either  underestimating  their  performance  or  were  very  accurate  in  predicting  their 
performance  (i.e.,  bias  scores  near  zero).  Results  are  discussed  in  the  context  of  metacognition 
in  general  and  adaptive  training  for  complex  tasks  in  particular. 

Fiore,  S.  M.,  Cuevas,  H.  M.,  Scielzo,  S.,  &  Salas,  E.  Metacognition  and  multimodalities: 
Altering  processes  and  products  in  distributed  learning.  Manuscript  in  preparation. 

In  this  experiment  we  investigated  the  degree  to  which  metacognition  can  be  altered  by 
hypermedia  training  manipulations.  We  looked  at  how  the  presentation  of  identical  information 
using  differing  modalities  might  alter  meta-comprehension  processes  and  subjective  workload. 
We  hypothesized  that  that  the  subjective  experience  (encoding  process)  may  be  altered  when 
attempting  to  leam  via  the  differing  presentation  manipulations.  As  such,  metacomprehension 
may  vary  across  these  modalities  dependent  upon  the  degree  to  which  participants  perceive  they 
have  successfully  retained  and/or  understood  the  material.  Because  audio  presentation  is  a  less 
typical  manner  of  information  acquisition,  participants  may  find  it  more  challenging  and  be  less 
confident  in  their  understanding  of  the  material.  Although  both  text  and  audio  are  thought  to  be 
processed  within  the  articulatory  loop,  our  argument  is  that  they  will  differently  interact  with 
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metacognitive  processes  (e.g.,  central  executive  processes)  thus  altering  overall  accuracy  in 
metacognition.  Thus,  participant  ratings  of  metacomprehension  may  be  lower  and  their 
assessment  of  workload  higher,  when  material  is  presented  auditorially.  These  predictions 
center,  not  on  knowledge  acquisition,  but  on  the  learner’s  perception  of  their  knowledge 
acquisition.  Although  there  may  be  little  or  no  difference  in  actual  performance  on  tests  of 
comprehension  dependent  upon  modality,  the  atypical  nature  of  the  encoding  experience  may 
alter  metacomprehension  prediction  accuracy. 


Scielzo,  S.,  Fiore,  S.  M.,  Cuevas,  H.  M.,  &  Salas,  E.  (2004).  Diagnosticity  of  mental  models  in 
cognitive  and  metacognitive  processes:  Implications  for  synthetic  task  environment 
training.  In  S.  G.  Schiflett,  L.  R.  Elliott,  E.  Salas,  &  M.  D.  Coovert  (Eds.),  Scaled 
worlds:  Development,  validation,  and  application  (pp.  181-199).  Aldershot,  UK: 
Ashgate. 

In  this  chapter  we  discuss  the  utility  of  mental  model  assessment  when  training  in  synthetic  task 
environments  (STE).  We  discuss  both  cognitive  and  metacognitive  processes  within  the  context 
of  knowledge  acquisition  and  describe  the  theoretical  issues  surrounding  these  factors  when 
training  for  complex  team  tasks.  We  additionally  report  a  portion  of  our  research  findings  in 
order  to  illustrate  how  mental  model  measurement  can  be  used  in  cognitively  diagnostic 
assessment.  This  study  used  an  automated  training  system,  designed  to  instruct  participants  to 
interact  within  a  synthetic  task  environment  for  military  command  and  control.  A  computer- 
based  knowledge  test  was  developed  to  assess  differing  components  of  knowledge  critical  to 
performance  in  this  STE  and  a  computerized  knowledge  elicitation  method  was  used  to  assess 
the  development  of  participants’  mental  models  when  training  for  this  task.  Additionally, 
metacognitive  processes  were  measured  prior  to,  and  following,  the  STE  knowledge  assessment 
battery.  We  show  how  mental  model  accuracy  is  indicative  of  both  accuracy  in  metacognitive 
processes  as  well  as  knowledge  acquisition  for  STE  training.  We  conclude  with  a  discussion  of 
the  implications  for  training  research  when  using  STEs. 


Supplemental  Work 

Cuevas,  H.  M.,  Fiore,  S.  M.,  Salas,  E.,  &  Bowers,  C.  A.  (2002).  A  macroergonomic  approach  to 
distributed  team  performance.  Proceedings  of  the  46th  Annual  Meeting  of  the  Human 
Factors  and  Ergonomics  Society,  1335-1339.  Santa  Monica,  CA:  HFES. 

Cuevas,  H.  M.,  Fiore,  S.  M.,  Salas,  E.,  &  Bowers,  C.  A.  (in  press).  Virtual  teams  as 
sociotechnical  systems.  To  appear  in  M.  Khosrow-Pour  (Ed.),  Encyclopedia  of 
Information  Science  and  Technology,  Volume  TV.  Hershey,  PA:  Idea  Group  Publishing. 

Fiore,  S.  M.,  Cuevas,  H.  M.,  Salas,  E.  &  Schooler,  J.  W.  (2002).  Distributed  teams  and 
distributed  memory.  Proceedings  of  the  46th  Annual  Meeting  of  the  Human  Factors  and 
Ergonomics  Society,  398-402.  Santa  Monica,  CA:  HFES. 
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Year  Three  -  Objectives  and  Accomplishments 

In  Year  Three  our  objectives  continued  to  involve  theoretical  and  empirical  thrusts  centered  on 
distributed  learning  and  training  with  an  increasing  emphasis  on  knowledge  application  for 
complex  tasks  when  distributed  learning  systems  are  utilized  (Cuevas,  2004).  Simultaneous  to 
this  we  have  continued  our  work  with  knowledge  acquisition  but  have  broadened  it  to  consider 
additional  methods  that  may  accelerate  the  knowledge  integration  process  when  training  for 
complex  tasks  (Cuevas,  Scielzo,  &  Fiore,  in  preparation;  Kring,  2004).  This  has  involved: 

1.  The  development  of  experiments  to  investigate  characteristics  of  the  training  to  determine 
how  instructional  science  strategies  may  be  adaptable  to  multiple  training  needs. 

2.  Examination  of  how  individual  differences  in  trainee  aptitudes  may  interact  with  the 
design  components  of  computer-based  training  on  complex  tasks. 

3.  The  team  process  factors  driven  by  team  cognition  and  which  may  alter  team 
coordination. 

Specifically  we  build  upon  the  research  assessing  the  degree  to  which  distributed  teams  and 
distributed  training  alter  individual  and  team  process  and  performance.  We  take  a  multi-pronged 
approach  to  converge  on  a  fuller  understanding  of  the  issues  surrounding  team  training  for 
complex  tasks  (e.g.,  metacognitive  processes  during  learning;  developing  integrated  knowledge 
via  concept  mapping)  along  with  the  task  and  environmental  characteristics  (task  complexity, 
high  workload)  that  influence  cognitive  processes  in  complex  task  domains  (e.g.,  memory  - 
dependent  actions)  (Cuevas,  Fiore,  &  Kring,  in  press;  Fiore,  Cuevas,  Schooler,  &  Salas,  in 
press).  We  have  similarly  considered  team  cognition  issues  both  within  and  across  teams  (Fiore, 
Jentsch,  Bowers,  &  Salas,  2003).  We  continued  with  assessing  how  these  factors  may  uniquely 
tax  individual  differences  in  aptitude  (e.g.,  verbal  ability)  in  order  to  determine  how  differing 
training  manipulations  may  scaffold  that  training  (Cuevas,  Fiore,  Bowers,  &  Salas,  2004; 
Cuevas,  Scielzo,  &  Fiore,  in  preparation). 

Last,  we  began  work  on  a  technology  transfer  component  of  our  research  effort.  This  involved 
the  creation  of  a  detailed  instructional  manual  describing  the  methodology  employed  in  the 
development  of  training  and  testing  modules  used  to  diagnose  differing  components  of 
knowledge  acquisition  (Scielzo,  Cuevas,  &  Fiore,  2004).  We  describe  via  narrative  and 
diagrammatic  presentation  the  manner  in  which  our  experimental  system  was  developed  so 
laboratories  and  government  entities  conducting  similar  research  can  use  this  methodology. 
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Year  Three  -  Key  Findings  and  Relevant  Products 
Theoretical  Work 


Bowers,  C.  A.  &  Salas,  E.  (Eds.),  (in  press).  Teams  and  technology.  Washington,  DC: 

American  Psychological  Association. 

One  of  the  most  striking  changes  in  modem  work  life  has  been  the  dramatic  increase  in  the 
importance  of  teamwork.  As  the  nature  of  work  grew  more  technical  and  complex,  it  became 
apparent  that  the  “one  person,  one  job”  approach  to  working  was  badly  outdated.  Companies 
scrambled  to  identify  ways  to  foster  teamwork  in  businesses  throughout  the  world.  Scientists 
scrambled  to  learn  about  the  nature  of  teamwork  while  managers  wrote  (and  bought)  hundreds  of 
books  on  the  subject.  The  impact  has  been  that  “teaming”  is  now  an  essential  part  of  how  almost 
every  company  in  the  world  does  business,  and  that  teamwork  is  considered  a  core  competency 
for  most  employees.  The  interest  in  teamwork  appears  to  be  fading,  somewhat,  because  most 
companies  have  it  “Figured  out.”  We  suggest,  however,  that  this  is  just  a  calm  before  another 
storm  involving  teams.  The  next  challenge  to  teamwork  in  organizations  is  to  cope  with  the 
effects  of  the  many  technologies  that  are  being  acquired  by  businesses.  This  is  no  small  matter. 
Most  of  what  we  know  about  teamwork  involves  groups  of  people  sitting  around  a  table. 
However,  modem  teams  rarely  sit  around  tables  anymore.  21st  century  teamwork  takes  place 
through  e-mail,  video  conferences,  collaborative  groupwork  systems  and  other  technologies. 
Furthermore,  teammates  are  not  necessarily  human.  Employees  now  get  advice  from  artificially 
intelligent  systems  that  function  as  “virtual  team  members.”  In  some  cases,  the  machine  is 
actually  “the  boss.”  Organizations  are  woefully  unprepared  to  integrate  these  technologies  into 
their  teams.  As  such,  in  this  volume  we  distill  the  best  and  most  current  scientific  knowledge 
and  combine  it  with  practical  “lessons  learned”  from  organizations  using  these  systems. 


Fiore,  S.  M.  &  Salas,  E.  (Eds.).  Where  is  the  learning  in  distance  learning?:  Towards  a  science 
of  distributed  learning  and  training.  Edited  volume  in  preparation.  Washington,  DC: 
American  Psychological  Association. 

Modifications  to  industrial  operations  and  rapid  advances  in  technology  have  created  an 
unparalleled  demand  for  training.  Simultaneous  to  this  has  been  a  rapid  advance  in  technology 
for  the  delivery  of  training.  Because  of  this,  traditional  “classroom”  learning  and  training 
approaches  have  been  increasingly  supplanted  by  distance-learning  efforts.  We  use  the  term 
“Distributed  Learning”  to  generally  describe  training  taking  place  while  the  student/trainee  is 
geographically  isolated  from  either  the  instructor  or  their  peers.  Although  this  technology  offers 
tremendous  promise,  and  is  already  being  widely  used  both  in  academia  and  industry  (e.g.,  over 
a  billion  dollars  is  typically  spent  on  non-classroom  delivery  of  training  annually),  the  scientific 
and  pedagogical  implications  of  such  instruction  are  unclear.  The  purpose  of  this  volume  is  to 
assemble  leading  researchers  whose  work  directly  bears  on  the  issues  critical  to  our 
understanding  of  the  theoretical  and  methodological  implications  surrounding  distributed 
learning. 
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Fiore,  S.  M.,  Cuevas,  H.  M.,  Schooler,  J.  &  Salas,  E.  (in  press).  Cognition,  teams,  and  team 
cognition:  Memory  actions  and  memory  failures  in  distributed  team  environments.  To 
appear  in  E.  Salas  &  C.  A.  Bowers  (Eds.),  Teams  and  technology.  Washington,  DC: 
American  Psychological  Association. 

The  implementation  and/or  use  of  distributed  teams  as  a  definable  organizational  unit  has 
dramatically  increased  the  structural  complexity  of  teams  in  organizations.  Adopting  a 
sociotechnical  systems  approach,  we  attempt  to  demonstrate  how  theories  from  cognitive  science 
can  be  used  to  explore  the  unique  challenges  emerging  from  this  new  organizational  structure. 
Specifically,  we  focus  on  the  memory  actions  that  are  foundational  to  effective  task  performance, 
particularly  in  these  complex  operational  environments,  and  highlight  how  the  team  opacity 
arising  from  distributed  interaction  can  impact  team  cognition,  potentially  leading  to  memory 
failures.  We  conclude  with  a  proposed  framework  designed  to  facilitate  our  understanding  of 
memory  failures  in  complex  operational  environments,  and  offer  guidelines  for  organizational 
practice. 


Fiore,  S.  M.,  Jentsch,  F.,  Bowers,  C.  A.,  &  Salas,  E.  (2003).  Shared  mental  models  at  the  intra- 
and  inter-team  level:  Applications  to  counter-terrorism  and  crisis  response  for  Homeland 
Security.  Proceedings  of  the  47th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics 
Society,  439-443.  Santa  Monica,  CA:  Human  Factors  and  Ergonomics  Society. 

Critical  to  meeting  the  needs  arising  from  an  increasing  emphasis  on  homeland  security  is  a 
fuller  understanding  of  how  to  maximize  the  interactions  among  a  diverse  group  of  people  and 
organizations.  The  complicating  issue  is  the  need  to  coordinate  experts  and  teams  of  experts 
from  differing  disciplines.  From  a  practical  standpoint,  because  of  the  multidisciplinary 
requirements  for  effective  homeland  security,  it  is  imperative  that  methodologies  that  can  foster 
inter-individual  (i.e.,  within  team)  and  inter-organizational  coordination  (i.e.,  across  departments 
or  organizations)  be  investigated.  From  a  theoretical  standpoint,  the  operational  concepts  that 
drive  the  investigation  and  development  of  such  methodologies  must  be  identified.  We  suggest 
that  the  concept  of  shared  mental  models,  arising  out  of  research  in  the  cognitive  and 
organizational  sciences,  can  be  utilized  in  a  program  of  research  that  explores  the  methods  and 
tools  to  facilitate  coordination  in  homeland  security.  Shared  mental  models  can  be  applied  to 
help  our  understanding  of  coordinated  behavior  at  both  the  team  level  and  the  organizational 
level  and,  in  this  paper,  we  illustrate  the  relevance  of  this  construct  to  both  counter-terrorism 
operations  and  crisis  response.  We  first  discuss  the  shared  mental  model  construct  and  follow 
this  with  a  brief  description  of  sample  applications. 


Fiore,  S.  M.,  Cuevas,  H.  M.,  &  Salas,  E.  (2003).  Putting  working  memory  to  work:  Integrating 
cognitive  science  theories  with  cognitive  engineering  research.  Proceedings  of  the  47 
Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society,  508-512.  Santa  Monica, 
CA:  Human  Factors  and  Ergonomics  Society. 

Human  factors  research  has  long  included  discussions  of  the  importance  of  mental  models  when 
understanding  human-human  interaction  and  human-system  interaction.  Additionally,  a  small 
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number  of  researchers  in  the  field  have  included  theories  of  working  memory  and  related 
constructs  in  their  research  approaches.  Nonetheless,  little  if  any  research  has  acknowledged  the 
connection  between  these  two  very  important  constructs  arising  out  of  the  cognitive  sciences.  In 
this  paper  we  discuss  recent  theoretical  and  empirical  developments  on  working  memory  from 
cognitive  psychology  and  cognitive  neuroscience  and  relate  these  to  our  understanding  of  mental 
models.  We  show  how  the  connection  between  these  two  concepts  can  facilitate  a  deeper 
understanding  of  issues  associated  with  cognitive  engineering  and  decision  making  research. 


Scielzo,  S.  &  Fiore,  S.  M.  A  two-stage  model  of  multimedia  learning:  Implications  for 
multimedia  design.  Manuscript  in  preparation. 

The  advances  of  multimedia  technology  supporting  computer-based  learning  environments  are 
such  that  the  research  necessary  to  understand  the  underlying  mechanisms  of  optimal  knowledge 
acquisition  did  not  coalesce  into  a  more  integrated  and  unified  approach.  In  order  to  assist 
multimedia  research  integration,  this  paper  first  reviews  various  factors  from  the  computer-based 
training  and  learning  literature  that  most  impact  knowledge  acquisition  while  introducing  the 
necessary  operationalization  of  various  concepts  to  help  tease  out  some  important  learning 
effects  generated  by  the  use  of  multimedia.  Second,  this  paper  proposes  a  two-stage  model  of 
multimedia  learning  based  on  classical  theoretical  models  and  recent  results  from  computer- 
based  multimedia  research  using  specific  terminology.  Finally,  stemming  from  this  model,  a 
series  of  propositions  are  brought  forth  to  foster  empirical  evaluation  of  the  model  while 
providing  guidelines  for  computer-based  multimedia  design. 


Technical  Work 


Scielzo,  S.,  Cuevas,  H.  M.,  &  Fiore,  S.  M.  (2004).  Dynamic  Distributed  Decision-Making: 
Development  of  mock-up  scenarios  for  synthetic  task  environment  training  (Technical 
Report  for  the  Air  Force  Office  of  Scientific  Research,  prepared  under  Grant  Number 
F49620-0 1-1-0214).  Orlando,  FL:  University  of  Central  Florida. 

In  this  technical  report,  we  first  report  the  methodological  procedures  used  to  develop  our 
training  tutorial,  composed  of  three  modules  (general  information,  procedural  training,  team 
training);  and  second,  we  report  the  development  of  our  knowledge  test,  composed  of  a 
declarative  and  an  integrative  section.  Specific  to  the  training.  Module  2  of  the  tutorial  contains 
several  videos  and  ‘mock-up’  scenarios  designed  to  mimic  the  environment  of  the  Dynamic 
Distributed  Decision-making  (DDD)  command-and-control  synthetic  task.  The  main  goal  of 
using  various  mock-up  scenarios  in  our  training  was  to  give  hands-on  experience  aimed  at 
refining  trainees’  procedural  skills.  As  such,  particular  cafe  was  devoted  to  artificially  render  the 
original  synthetic  task  environment  (STE).  The  mock-up  scenarios  were  developed  with  two 
design  characteristics:  1)  Mock-up  scenarios  faithfully  reproduced  the  original  environment  for 
accurate  transfer  of  procedural  skills;  2)  Visual  feedback  was  incorporated  in  the  mock-up 
scenarios  in  order  to  guide  the  trainee  into  effectively  learning  specific  procedures.  In  the 
knowledge  test,  particular  attention  was  given  to  the  ‘integrative’  portion,  consisting  of  animated 
videos  of  specific  DDD  team  scenarios.  Those  scenarios  were  developed  to  test  trainees’ 
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knowledge  transfer  of  the  game  to  team  related  questions.  For  both  the  training  tutorial  and  the 
integrative  portion  of  the  knowledge  test,  Microsoft®  PowerPoint®  2000  and  TechSmith® 
Snaglt®  were  employed.  The  two  software  programs  were  used  in  combination  to  integrate 
screen  captures  and  video  captures  of  the  DDD  environment  into  self-paced  presentations. 


Empirical  Work 

Cuevas,  H.  M.,  Fiore,  S.  M.,  Bowers,  C.  A.,  &  Salas,  E.  (2004).  Fostering  constructive  cognitive 
and  metacognitive  activity  in  computer-based  complex  task  training  environments 
(Invited  paper  for  Special  Issue  on  Human-Computer  Interaction).  Computers  in  Human 
Behavior,  20, 225-241. 

In  this  paper,  we  describe  a  multi-year  programmatic  research  effort  aimed  at  investigating  the 
use  of  interactive  computer-based  training  technology,  such  as  enhanced  displays,  to  support  the 
acquisition,  development,  and  transfer  of  knowledge  related  to  critical  linkages  in  domain 
knowledge  for  complex  task  training  environments  (e.g.,  aviation,  distributed  decision-making). 
We  also  introduce  preliminary  findings  from  our  most  recent  work  examining  the  effectiveness 
of  guided,  learner-generated  instructional  strategies  embedded  within  computer-based  training, 
designed  to  prompt  learners  to  monitor  their  comprehension  and  mindfully  engage  in 
constructive  cognitive  and  metacognitive  activities.  We  review  the  successive  iterations  of  our 
investigation,  and,  based  upon  the  significant  findings  in  our  studies,  extract  a  set  of  propositions 
for  the  design  and  evaluation  of  computer-based  training  programs  for  complex  systems.  These 
propositions  will  be  presented  in  the  context  of  the  related  theoretical  rationale  drawn  from  the 
literature  and  the  relevant  empirical  support  from  the  body  of  research  conducted  over  these  past 
three  years. 


Scielzo,  S.,  Fiore,  S.  M.,  &  Cuevas,  H.  M.  Multimedia  design  and  redundancy  in  complex 
training  environments.  Manuscript  in  preparation. 

The  present  study  investigated  how  manipulating  the  redundancy  state  (simple  or  complex 
redundancy)  of  multimedia-based  instruction  may  impact  knowledge  acquisition,  and  how 
distinct  training  effectiveness  measures  may  be  best  used  to  evaluate  multimedia  design 
principles  as  they  apply  to  training  for  complex  systems.  The  results  show  that  redundancy  of 
information,  which  has  previously  been  shown  to  negatively  impact  knowledge  acquisition, 
yielded  the  opposite  results  in  our  study.  These  results  suggest  that  high  overlay  of  redundant 
codes  of  information  can  foster  knowledge  retention.  A  similar  pattern  of  results  was  found  in 
terms  of  reduced  cognitive  load,  as  indicated  by  higher  instructional  efficiency  achieved  via  the 
redundant  training  program.  The  discussion  centers  on  the  generalizability  of  principles  of 
multimedia  learning  when  applied  to  more  complex  training  environments.  Implications  for 
multimedia  design  are  discussed  within  the  context  of  cognitive  load  theory. 
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Fiore,  S.  M.,  Scielzo,  S.,  Cuevas,  H.  M.,  &  Salas,  E.  Working  with  working  memory  in  complex 
task  training.  Manuscript  in  preparation. 

Foundational  to  our  understanding  of  the  successful  implementation  of  distributed  training  is  the 
investigation  of  the  impact  that  individual  differences  in  abilities  have  on  the  cognitive  processes 
and  cognitive  products  emerging  from  these  environments.  In  order  to  accomplish  this  objective 
we  need  reliable  and  valid  instruments  to  assess  learners’  cognitive  abilities.  Kyllonen  and 
colleagues  (1993,  1994)  developed  the  Cognitive  Abilities  Measurements  (CAM)  framework  to 
maximize  the  individual  differences  variability  in  performance  accounted  for  using  assessment 
tests.  CAM  is  an  evolution  of  a  4-factor  model,  emphasizing  the  significant  role  of  individual 
differences  in  working  memory  capacity,  information-processing  speed,  breadth  of  declarative 
knowledge,  and  breadth  of  procedural  knowledge  (see  also,  Kyllonen  &  Christal,  1989).  The 
CAM  battery  can  serve  as  a  reliable,  valid,  and  powerful  tool  to  measure  individual  differences 
in  cognitive  abilities  that  are  deemed  critical  for  successful  learning  outcomes  in  complex  task 
training  (Kyllonen,  1993).  Furthermore,  of  the  many  cognitive  ability  factors  evaluated  by  the 
CAM  battery  that  seem  to  predict  individual  variability  in  performance,  working  memory  tests 
were  found  to  be  “maximally  valid,  that  is,  they  correlate[d]  more  highly  with...  learning 
criterion[s]  than  any  other  kind  of  test”  (Kyllonen,  1994,  p.  336).  Therefore,  the  CAM  battery 
can  be  effectively  utilized  in  our  research  effort  to  evaluate  the  extent  to  which  individual 
differences  in  working  memory  capacity  affect  cognitive  processes  and  cognitive  products  in 
distributed  learning  environments.  Thus  far  we  have  updated  CAM-4  for  use  in  a  Windows 
environment  (from  a  DOS-based  program).  In  a  pilot  study  we  have  successfully  used  this 
updated  battery  and  we  are  currently  analyzing  how  these  individual  differences  in  working 
memory  interact  with  complex  task  training. 


Kring,  E.,  D.  (2004).  Improving  metacomprehension  and  learning  through  graduated  concept 
model  development  (Doctoral  dissertation,  University  of  Central  Florida,  Orlando,  2004). 

Mental  model  development,  deeper  levels  of  information  processing,  and  elaboration  are  critical 
to  learning.  More  so,  individuals’  metacomprehension  accuracy  is  integral  to  making 
improvements  to  their  knowledge  base.  In  other  words,  without  an  accurate  perception  of  their 
knowledge  on  a  topic,  learners  may  not  know  that  knowledge  gaps  or  misperceptions  exist  and, 
thus,  would  be  less  likely  to  correct  them.  Therefore,  this  study  offered  a  dual-process  approach 
that  aimed  at  enhancing  metacomprehension.  One  path  aimed  at  advancing  knowledge  structure 
development  and,  thus,  mental  model  development.  The  other  focused  on  promoting  a  deeper 
level  of  information  processing,  through  processes  like  elaboration.  It  was  predicted  that  this 
iterative  approach  would  culminate  in  improved  metacomprehension  and  increased  learning. 

Accordingly,  using  the  Graduated  Concept  Model  Development  (GCMD)  approach,  the  role  of 
learner-generated  concept  model  development  in  facilitating  metacomprehension  and  knowledge 
acquisition  was  examined.  Concept  maps  have  had  many  roles  in  the  learning  process  as  mental 
model  assessment  tools  and  advanced  organizers.  However,  this  study  proposes  that  the  process 
of  concept  model  building  may  also  be  an  effective  training  tool.  Where  advanced  organizers 
likely  rely  on  recognition,  concept  model  development  requires  recalling  and  understanding 
‘how’  and  ‘why’  the  interrelationships  between  concepts  exist.  Indeed,  central  to  improving 


16 


Salas,  Fiore,  and  Bowers  (UCF)  Final  Report  17 


knowledge  deficiencies  and  misunderstandings  is  metacomprehension,  and  the  constructing  of 
concepts  maps  is  likely  to  improve  metacomprehension  accuracy  and,  thus,  learning. 

The  GCMD  alone  condition  showed  a  stronger  indication  of  metacomprehension  accuracy,  via 
prediction  measures,  compared  with  the  other  three  conditions  (Control,  Advanced  Organizer, 
and  GCMD  with  Feedback),  and,  specifically,  significantly  higher  correlations  than  the  other 
three  conditions  in  declarative  knowledge.  Self-efficacy  measures  also  indicated  that  the  higher 
metacomprehension  accuracy  correlation  observed  in  the  GCMD  condition  was  likely  the  result 
of  the  intervention,  and  not  due  to  differences  in  self-efficacy  in  that  group  of  participants. 
Likewise,  the  Feedback  and  GCMD  conditions  led  to  significantly  high  correlations  for 
metacomprehension  accuracy  based  on  levels  of  understanding  on  declarative  knowledge  tutorial 
module.  The  Feedback  condition  also  showed  similar  findings  for  the  strategic  knowledge 
module. 

Results,  however,  failed  to  reveal  any  significant  differences  between  the  four  conditions  in 
mental  model  accuracy  or  performance  on  a  knowledge  assessment  task.  Nevertheless,  results 
do  support  the  relevance  of  accurate  mental  model  development  in  knowledge  assessment 
outcomes.  Specifically,  in  terms  of  cognitively  diagnostic  assessment,  results  showed  that 
mental  model  accuracy  (as  measured  by  similarity  to  an  expert  model)  was  predictive  of 
knowledge  acquisition.  Greater  similarity  to  an  expert  model  led  to  significantly  greater 
performance  on  measures  of  basic  factual  knowledge  (declarative  questions),  and  more 
importantly,  measures  assessing  integration  and  application  of  task-relevant  knowledge 
(integrative  questions). 

Retrospectively,  two  opposing  factors  my  have  led  to  some  insignificant  findings.  From  one 
side,  the  experimental  measures  may  not  have  been  rigorous  enough  to  filter  out  the  effect  from 
the  intervention  itself.  Conversely,  software  usability  issues  and  the  resulting  limitations  in 
experimental  design  worked  negatively  against  the  two  concept  mapping  conditions  and, 
inadvertently,  suppressed  effects  of  the  intervention.  Future  research  in  the  GCMD  approach 
will  likely  review  cognitive  workload,  concept  mapping  software  design,  and  the  sensitivity  of 
the  measures  involved. 


Cuevas,  H.  M.,  Scielzo,  S.,  &  Fiore,  S.  M.  Evaluating  the  instructional  efficiency  of  a  guided 
learner-generated  questioning  strategy  within  computer-based  complex  training 
environments.  Manuscript  in  preparation. 

This  study  assessed  the  extent  to  which  a  guided  learner-generated  questioning  strategy  (learning 
check)  could  facilitate  acquisition  of  task-relevant  knowledge  and  improve  the  instructional 
efficiency  (i.e.,  relation  between  perceived  cognitive  effort  during  training  and  testing  and  post¬ 
training  performance)  for  a  complex  dynamic  distributed  decision-making  task.  The  learning 

checks  involved  the  use  of  a  content-free  question  stems  (e.g.,  “How  are _ and _ 

related?”),  designed  to  prompt  trainees  to  actively  monitor  their  own  learning.  In  addition,  this 
study  also  investigated  the  degree  to  which  individual  differences  in  learner  aptitudes  (verbal 
comprehension  ability)  interacted  with  the  instructional  strategy  to  impact  training  outcomes.  In 
general,  the  results  highlighted  the  important  role  of  individual  differences  in  relevant  learner 
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aptitudes  for  achieving  successful  outcomes  in  computer-based  training  environments. 
Participants  demonstrating  high  verbal  comprehension  ability  consistently  outperformed 
participants  with  low  verbal  comprehension  ability  on  the  knowledge  assessment  task.  Further, 
results  also  showed  that  the  training  program’s  instructional  efficiency  was  significantly  greater 
when  evaluated  for  the  high  verbal  comprehension  ability  participants,  as  compared  to  the  low 
verbal  comprehension  ability  participants.  In  addition,  incorporating  the  learning  check  strategy 
into  the  instruction  also  consistently  yielded  positive  instructional  efficiency  scores  overall. 
Finally,  with  regard  to  aptitude-treatment  interactions,  findings  showed  that  incorporating  the 
learning  check  strategy  into  the  training  significantly  improved  the  program’s  instructional 
efficiency  when  examining  the  results  for  low  verbal  comprehension  ability  participants.  This 
beneficial  effect  on  the  training  program’s  instructional  efficiency  was  not  evident  when  results 
were  examined  for  high  verbal  comprehension  ability  participants.  Results  are  discussed  in 
terms  of  the  implications  for  the  design  of  adaptive  learning  systems. 


Cuevas,  H.  M.,  Fiore,  S.  M.,  Bowers,  C.  A.,  &  Salas,  E.  (2004).  Using  guided  learner-generated 
instructional  strategies  to  transform  learning  into  a  constructive  cognitive  and 
metacognitive  activity.  Proceedings  of  the  48th  Annual  Meeting  of  the  Human  Factors 
and  Ergonomics  Society,  1049-1053.  Santa  Monica,  CA:  HFES. 

Successful  learning  outcomes  in  learner-controlled,  computer-based  training  environments  are 
inherently  dependent  upon  learners’  possession  of  well-developed  metacognitive  skills,  that  is, 
how  well  learners  are  able  to  accurately  monitor  and  regulate  their  knowledge  acquisition 
process.  The  present  study  explored  the  effectiveness  of  embedding  a  guided  learner-generated 
instructional  strategy  (query  method),  designed  to  support  learners’  cognitive  and  metacognitive 
processes  within  the  context  of  computer-based  complex  task  training.  The  queries  were 
presented  as  “stop  and  think”  exercises  in  an  open-ended  question  format  that  asked  learners  to 
generate  complex  (high  level  elaboration)  sentences  from  a  list  of  key  concepts  presented  in  the 
training.  In  terms  of  cognitive  processes,  results  showed  that  incorporating  the  query  method 
into  the  training  resulted  in  improved  integration  and  application  of  task-relevant  knowledge,  as 
indicated  by  greater  similarity  to  an  expert  model  of  the  domain  and  better  performance  on 
integrative  knowledge  assessment  requiring  the  integration  and  application  of  task-relevant 
concepts.  With  regard  to  metacognitive  processes,  the  query  method  may  have  also  assisted 
participants  in  more  accurately  monitoring  their  comprehension,  as  indicated  by  their 
significantly  lower  bias  scores.  Results  are  discussed  in  the  context  of  designing  adaptive 
learning  systems. 


Cuevas,  H.  M.  (2004).  Transforming  learning  into  a  constructive  cognitive  and  metacognitive 
activity:  Use  of  a  guided  learner-generated  instructional  strategy  within  computer-based 
training  (Doctoral  dissertation,  University  of  Central  Florida,  Orlando,  2004). 

The  present  study  further  explored  the  effectiveness  of  embedding  a  guided,  learner-generated 
instructional  strategy  (query  method),  designed  to  support  learners’  cognitive  and  metacognitive 
processes  in  computer-based  training  environments.  The  queries  were  presented  as  “stop  and 
think”  exercises  in  an  open-ended  question  format  that  asked  learners  to  generate  either  simple 
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(low  level  elaboration)  or  complex  (high  level  elaboration)  sentences  from  a  list  of  key  concepts 
presented  in  the  training.  Overall,  results  consistently  highlighted  the  beneficial  effect  of 
presenting  participants  with  low-level  elaboration  queries,  as  compared  to  the  no-query  or  high- 
level  elaboration  query  conditions. 

In  terms  of  post-training  cognitive  outcomes,  participants  presented  with  the  low-level 
elaboration  queries  generated  significantly  more  accurate  sentences,  exhibited  significantly  more 
accurate  knowledge  organization  (as  indicated  by  greater  similarity  to  an  expert  model),  better 
acquisition  of  perceptual  knowledge,  and  superior  performance  on  integrative  knowledge 
assessment  requiring  the  integration  and  application  of  task-relevant  concepts.  Consistent  with 
previous  studies,  no  significant  differences  in  performance  were  found  on  basic  factual 
knowledge  assessment.  Presentation  of  the  low-level  elaboration  queries  also  significantly 
improved  the  training  program’s  instructional  efficiency,  that  is,  greater  performance  was 
achieved  with  less  perceived  cognitive  effort.  In  terms  of  post-training  metacognitive  outcomes, 
participants  presented  with  the  low-level  elaboration  queries  exhibited  significantly  greater 
metacomprehension  accuracy  and  more  effective  metacognitive  self-regulation  during  training. 

The  results  of  this  study  are  discussed  in  terms  of  the  theoretical  implications  for  gamering  a 
better  understanding  of  the  cognitive  and  metacognitive  factors  underlying  the  learning  process. 
Practical  implications  for  training  design  are  presented  within  the  context  of  cognitive  load 
theory  and  the  need  for  a  multi-faceted  approach  to  training  evaluation. 

Supplemental  Work 

Cuevas,  H.  M.,  Fiore,  S.  M.,  &  Kring,  J.  P.  (in  press).  Understanding  memory  actions  and 
memory  failures  in  extreme  environments.  Proceedings  of  the  Second  Annual  Meeting  of 
the  Society  for  Human  Performance  in  Extreme  Environments  (HPEE).  Orlando,  FL: 
HPEE. 
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BENEFITS  TO  AFOSR 

The  benefits  to  AFOSR  of  this  research  effort  fit  within  the  overall  mission  of  understanding 
human  information  processing  within  complex  environments.  Specifically,  a  portion  of  the 
overarching  objectives  of  AFOSR  research  on  human  performance  involve  the  development  of 
models  and  methods  associated  with:  (1)  the  cognitive  factors  related  to  the  acquisition  of 
complex  skills;  (2)  the  individual  attributes  impacting  human  performance;  and,  (3)  team 
performance  in  complex  environments.  In  this  section  we  briefly  highlight  how  the  findings 
from  this  grant  help  AFOSR  meet  these  objectives  and  follow  this  with  an  example  application 
to  Air  Force  operations. 

Cognitive  Factors  Related  to  the  Acquisition  of  Complex  Skills 

We  have  illustrated  how  distributed  learning  based  methods  of  training  can  be  used  to  quantify 
potential  performance  by  diagnosing  the  cognitive  factors  related  to  skill  acquisition  associated 
with  complex  distributed  team  tasks.  These  findings  aid  the  Air  Force  by  providing  the 
foundation  on  which  we  can  understand  the  learning  processes  associated  with  transforming  the 
novice  to  the  expert  as  well  as  the  training  methodologies  to  scaffold  that  transformation. 

•  We  have  developed  methodologies  that  assess  one’s  ability  to  describe  and  explain 
components  of  one’s  mental  models  associated  with  a  complex  distributed  team  task. 

•  We  have  devised  a  dynamic  measure  using  animated  scenario  vignettes.  These  assess  one’s 
ability  to  accurately  combine  differing  knowledge  components  and  predict  future  events. 

•  We  have  shown  the  utility  of  using  knowledge  elicitation  techniques  to  gauge  knowledge 
acquisition  by  documenting  a  relation  between  performance  and  accurate  mental  model 
development. 

•  We  have  documented  how  differing  forms  of  information  presentation  (i.e.,  audio  versus 
text)  can  alter  the  cognitive  processes  associated  with  training  for  complex  tasks. 

•  Based  upon  the  aforementioned  methods  of  cognitively-based  diagnoses,  we  have  discussed 
how  training  strategies  associated  with  knowledge  acquisition  and  knowledge  integration  can 
be  developed. 

•  We  have  developed  an  augmented  model  of  multimedia  learning  where  factors  associated 
with  differing  modalities  of  presentation  are  accounted  for  in  training  outcomes  for  complex 
tasks. 

•  We  have  compiled  an  edited  volume  on  distributed  and  distance  training  where  leading 
researchers  in  the  field  discuss  their  theory  and  findings  in  this  area. 

Individual  Attributes  Impacting  Human  Performance 

We  have  begun  to  identify  and  quantify  the  individual  attributes  related  to  learning  and 
performance  in  complex  environments.  These  findings  aid  the  Air  Force  by  providing  the 
foundation  on  which  to  understand  how  to  develop  training  systems  that  are  adaptable  to 
differing  characteristics  of  learners  and  teams  of  learners.  Thus,  we  have  identified  a  portion  of 
the  basic  issues  associated  with  the  development  of  systems  that  will  account  for  diversity  in 
aptitudes,  skills,  and  knowledge. 

•  We  have  documented  how  verbal  ability  differentially  influences  the  cognitive  processes 
associated  with  predicting  one’s  mastery  of  material  for  complex  task  training.  Further,  we 
have  shown  how  the  more  complex  the  measure,  the  greater  the  inaccuracy  for  low  verbal 
participants  and  the  greater  the  accuracy  for  high  verbal  participants. 
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•  We  have  illustrated  how  general  predispositional  traits  associated  with  metacognition  can  be 
predictive  of  performance  when  training  with  synthetic  team  tasks 

•  We  have  begun  preliminary  investigation  of  how  the  working  memory  subcomponent  of  the 
Air  Force’s  “Cognitive  Abilities  Measure”  is  related  to  performance  when  training  for 
complex  tasks. 

Team  Performance  in  Complex  Environments 

Via  theoretical  analyses  we  have  articulated  the  critical  components  associated  with  learning  and 
performance  in  distributed  team  tasks.  These  findings  benefit  the  Air  Force  by  providing  the 
foundation  on  which  to  understand  how  the  paucity  of  task  and/or  team  artifacts  associated  with 
distributed  interaction  can  influence  team  process  and  performance,  and  subsequently,  how 
training  programs  can  be  devised  to  attenuate  this  impact. 

•  We  have  described  how  distributed  interaction  may  alter  the  workload  associated  with  team 
interaction  and  thus  have  a  cascading  effect  on  team  coordination 

•  We  have  discussed  how  distributed  interaction  can  impact  the  attitudinal  factors  associated 
with  team  process  and  subsequently  alter  team  performance 

•  We  have  articulated  how  distributed  interaction  may  impinge  upon  or  alter  the  memory 
processes  associated  with  complex  team  tasks  and  thus  hinder  team  coordination. 

•  We  have  completed  an  edited  volume  providing  an  avenue  with  which  researchers  from  a 
diverse  array  of  fields  can  present  their  theories  and  findings  on  team  cognition  in  one 
integrated  volume.  This  book  allows  researchers  from  differing  disciplines  to  be  brought  up- 
to-date  on  both  theoretical  and  methodological  issues  surrounding  team  cognition. 

•  We  are  completing  an  edited  volume  on  teams  and  technology  where  researchers  discuss  the 
many  ways  team  process  and  performance  is  altered  by  technological  innovation. 


Sample  Air  Force  Applications 

A  number  of  issues  are  driving  current  military  needs  in  basic  research  leading  to  emerging 
Defense  Transformation  goals.  The  underlying  cause  of  this  need  for  transformation  is  the 
dramatic  impact  computer  and  information  technology  has  had  on  training.  Within  this  context 
we  focused  our  research  to  address  these  changed.  In  particular,  due  to  technological  advances, 
the  last  few  years  have  witnessed  a  tremendous  increase  in  machine-delivered  training. 
Computer-based  training  tools  appear  to  offer  a  number  of  advantages  over  traditional  methods: 
they  are  relatively  inexpensive,  they  can  be  used  in  the  field,  and  they  can  be  programmed  to 
adapt  to  the  learner’s  performance.  However,  while  this  technology  offers  tremendous  promise, 
it  is  clear  that  our  understanding  of  computer-based  instruction  is  currently  insufficient  to  use 
this  technology  effectively  in  many  situations  and  to  predict  its  efficacy  and  efficiency  across 
training  contents  and  situations.  Furthermore,  today’s  digital-age  soldiers  represent  a  challenge 
to  military  training  doctrine  in  that  their  life  experiences  are  borne  out  of  the  computer  age.  This 
has  produced  a  need  for  understanding  of  how  to  most  efficaciously  present  training  material  to 
this  new  group  of  recruits.  The  confluence  of  these  events  produces  the  need  for  a  programmatic 
basic  research  effort  that  can  disentangle  the  fundamental  issues  surrounding  learning  and 
performance  in  the  networked  battlefield. 

As  an  example  of  modem  training  needs,  current  Air  Force  operations  such  as  AW  ACS 
Weapons  Director  teams  have  to  manage  complex  distributed  team  interactions.  These 


21 


Salas,  Fiore,  and  Bowers  (UCF)  Final  Report  22 


operational  environments  have  a  dynamic  mix  of  cognitive  and  socio-technical  factors  that 
impact  performance.  These  range  from  synchronous  interaction  in  the  absence  of  co-location,  to 
the  multi-sensory  nature  of  systems  that  require  the  integration  of  a  variety  of  perceptual  and 
conceptual  knowledge.  Information  is  received  from  a  number  of  audio  and  visual  sources  and 
this  information  must  be  relayed  and/or  interpreted  while  communicating  with  others.  In  order 
to  avoid  problems  associated  with  identification  errors,  disorientation,  and  general  failures  in 
situation  awareness,  operators  in  such  environments  must  have  well-developed  mental  models. 
These  mental  models  consist  of  appropriately  interlinked  concepts  associated  with  their  task  and 
task  environment.  Further,  given  the  team  component  of  their  task,  individuals  in  AW  ACS 
teams  heavily  rely  upon  share  mental  models  of  the  operational  environment. 

Our  effort  is  applicable  to  a  number  of  these  issues  and  we  address  these  problems  by  analyzing 
human  performance  on  multiple  levels.  Generally,  in  order  to  appropriately  investigate  such 
complex  phenomena,  and  integrate  methodologies  and  issues,  a  multi-level  theoretical  approach 
is  required.  We  have  theorized  about  multiple  levels  of  analysis  (i.e.,  individual  factors  and  team 
factors)  in  order  to  better  specify  how  we  conceptualize  the  complex  interaction  among 
constructs.  For  example,  we  have  articulated  how  information  integration  supports  successful 
performance  in  dynamic  team  tasks  (e.g.,  Fiore,  Cuevas,  Scielzo,  &  Salas,  2002;  Scielzo,  Fiore, 
Cuevas,  &  Salas,  2004).  Additionally,  we  have  described  how  the  lack  of  co-location  alters  both 
cognitive  factors  as  well  as  subsequent  team  development  factors,  all  of  which  impact  team 
coordination  (e.g.,  Cuevas,  Fiore,  Salas,  &  Bowers,  2004;  Fiore,  Salas,  Cuevas,  &  Bowers, 
2003).  In  short,  our  effort  articulates  how  these  environments  can  impinge  upon  the  team 
processes  essential  to  effective  coordination.  And,  from  the  cognitive-level,  we  describe  how  the 
integration  of  critical  knowledge  components  associated  with  these  tasks  can  attenuate  these 
effects. 


CONCLUSION 

In  the  context  of  learning  complex  tasks,  distributed  training  may  prove  more  cost-effective 
because  of  the  dynamic  task  environments  in  which  Air  Force  personnel  must  train  and  work. 
Specifically,  distributed  training  designed  to  facilitate  the  acquisition  of  knowledge  that  is  both 
rich  and  flexible  affords  one  the  ability  to  accommodate  new  and  complex  technology  relatively 
easily.  The  aforementioned  findings  provide  the  foundation  on  which  the  Air  Force  training 
community  may  determine  how  distributed  systems  can  be  best  designed  to  account  for  diversity 
in  trainee  aptitude  and  disposition.  Overall,  the  outcomes  of  our  effort  support  the  AFOSR 
Mission  of  providing  the  best  training  tools  to  ensure  that  the  warfighter  develops  the  skills  to 
win.  By  utilizing  an  approach  that  encompasses  the  organizational  and  cognitive  sciences,  we 
have  provided  foundational  research  to  the  methods  for  the  development  and  evaluation  of 
warfighter  training  at  the  individual  and  team  levels.  These  include  the  tools  and  technologies 
based  upon  data-driven  empirical  efforts  and  our  theoretically-driven  organizing  frameworks. 
More  generally,  this  research  is  foundational  to  Air  Force  attempts  to  identify  the  type(s)  of 
knowledge  and  learning  strategies  most  useful  for,  and  most  generalizable  across,  complex  tasks 
and  how  distributed  learning  systems  may  facilitate  the  development  of  such  knowledge  for 
individuals  and  teams. 
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PERSONNEL  SUPPORTED  and/or  ASSOCIATED 

Principle  Investigators 

•  EDUARDO  SALAS,  Ph.D.,  is  a  Professor  of  Psychology  at  the  University  of  Central 
Florida  and  Program  Director  and  Professor  for  Human  Systems  Integration  Research 
Department  at  the  Institute  for  Simulation  and  Training 

•  STEPHEN  M.  FIORE,  Ph.D.,  is  Director  of  the  Consortium  for  Research  in  Adaptive 
Distributed  Learning  Environments,  a  laboratory  that  emphasizes  interdisciplinary  research 
incorporating  aspects  of  cognitive,  social,  and  organizational  psychology  in  the  investigation 
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